Repository navigation
Conversation
…yResource Since NVIDIA#3405, PinnedMemoryResource.__init__ without options calls cuMemPoolSetAccess to grant the current device read/write access to the driver's default pool. The driver refuses that call, and invalidates the capture, while a global-mode capture is active on any thread. Users of PyTorch's torch.cuda.graph (which defaults to global mode) therefore see CUDA_ERROR_STREAM_CAPTURE_UNSUPPORTED on construction and CUDA_ERROR_STREAM_CAPTURE_INVALIDATED on end_building. Fix: introduce MP_set_access in _memory_pool.pyx, wrapping cuMemPoolSetAccess in the existing _enter_relaxed_capture_mode / _exit_relaxed_capture_mode helpers, following the identical pattern used by pool attribute reads and writes and trim() since NVIDIA#2834. Call it from _PMR_init instead of the bare with-nogil block. The with-options path (cuMemPoolCreate) is unaffected; that call is still refused under global capture as documented in the 1.3.0 notes. Add tests for the three capture combinations (global+same_thread, global+other_thread, thread_local+same_thread) and a capture-mode restore check modelled on test_dmr_init_restores_capture_mode. Closes NVIDIA#3451
Contributor
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
closes #3451
Since #3405,
PinnedMemoryResource.__init__without options callscuMemPoolSetAccessto grant the current device read/write access to the driver's default host pool. The CUDA driver refuses that call, and invalidates the capture, while a global-mode capture is active on any thread. This breaks code that constructsPinnedMemoryResource()while another thread holds an active global-mode graph capture — including PyTorch'storch.cuda.graph, which uses global mode by default.The fix introduces
MP_set_accessin_memory_pool.pyx, wrappingcuMemPoolSetAccessin the existing_enter_relaxed_capture_mode/_exit_relaxed_capture_modehelpers. This is the same pattern already used for pool attribute reads and writes andtrim()since #2834._PMR_initcalls the new helper instead of the barewith nogil: HANDLE_RETURN(...)block.The with-options path (
cuMemPoolCreate) is intentionally unaffected; that call remains refused under global capture as documented in the 1.3.0 notes.Files changed:
cuda_core/cuda/core/_memory/_memory_pool.pxd— declareMP_set_accesscuda_core/cuda/core/_memory/_memory_pool.pyx— implementMP_set_accesswith relaxed-capture wrappercuda_core/cuda/core/_memory/_pinned_memory_resource.pyx— callMP_set_access; remove now-unusedas_cucimport; add one-sentence docstring notecuda_core/cuda/core/_memory/_pinned_memory_resource.pyi— stub regenerated by pre-commitcuda_core/tests/test_memory.py— addtest_pmr_construction_during_capture_keeps_capture_valid(three capture combinations) andtest_pmr_construction_restores_capture_mode, modelled on the existingtest_mempool_calls_during_capture_keep_capture_validandtest_dmr_init_restores_capture_modecuda_core/docs/source/release/1.3.1-notes.rst— new release notes file with fix entryCounterpart review (AGENTS.md dual-root requirement): This change is entirely in
cuda_core/, which is single-sourced. Thecuda_bindings_12/cuda_bindings_13roots are unaffected;cuMemPoolSetAccesswrappers there are unchanged and no behavioral change applies to them.SYNCED blocks: No SYNCED blocks are touched by this diff.
Checklist